The present invention is related to computer primary data storage systems and methods that provide comprehensive data protection. As background to the invention, hierarchical storage management (HSM) is a data management technique that is deployed with multiple types of data storage technologies like magnetic disk drives, magnetic tape drives, and optical disk drives. HSM has been used to transparently move the contents of files that are least recently accessed by client applications from higher speed, higher cost magnetic disk technology to lower cost, lower speed media like magnetic tape and optical disk.
These various storage technologies can be arranged in a multi-level hierarchy from fastest to slowest and/or from most costly to least costly. For the following example, assume that magnetic disks are employed as the fast, more costly primary storage device and magnetic tape is employed as the lower cost, slower access storage technology within an HSM hierarchy. With HSM in operation, the files that are most-recently accessed by client applications are retained in their complete form on magnetic disk and least recently accessed files are migrated out to magnetic tape. HSM software enables client applications to transparently access files that have been migrated out to magnetic tape. When a file that is least-recently used has been migrated out to magnetic tape, a much smaller stub file remains on magnetic disk, typically 1 KB to 4 KB in size. This stub file contains all of the basic operating system file information as well as an indication that this file has been migrated from the magnetic disk to another medium. The stub file also includes information that HSM uses to acquire the entire contents of a file from a magnetic tape and restore the complete content of a file to magnetic disk transparently when it has been requested by a client application.
Thus, HSM can be an effective way to manage large amounts of infrequently accessed data in a cost effective manner. HSM is also effective in eliminating the administrative alerts and subsequent operational actions associated with filesystems running out of available capacity for additional client data. Without HSM, when a filesystem runs out of available capacity, a storage administrator must quickly employ one of the following manual data management techniques to allow client applications to remain operational:                Use a third-party archiving program to identify unwanted files and commit these to magnetic tape or optical disk. Archiving deletes all information about files from the fileserver's filesystem and applications cannot access these files without having them recovered from archive tapes or optical disks.        If the fileserver can continue to accommodate additional magnetic disk drives, these are added to the server and associated volume and filesystem are expanded. This is a time consuming process and may result in client application downtime.        If a fileserver has been expanded to its limits of disk storage capacity, the administrator must migrate some number of shares of data (a share being a directory or folder of storage capacity) to another server that has available capacity.Contrasting this with HSM-based fileservers, as filesystems fill up, least recently used data is automatically migrated out to another storage medium like tape. Unlike archiving, HSM provides client applications with transparent continual access to their files, even when they've been migrated to magnetic tape. There is no need to add additional magnetic disk storage to a fileserver that supports HSM since storage is expanded by adding more magnetic tapes to the lower-level in the storage hierarchy. Finally, migrating shares from one server to another does not have to be performed specifically for reasons of balancing capacity across multiple fileservers.        
In U.S. Pat. No. 5,537,585, Blickenstaff, et al. describe an HSM system that migrates least recently used data from a fileserver to a lower-cost back-end removable storage system. The HSM servers are monitored for filesystem capacity utilization, and when they fill up, least recently used files are identified and migrated out to back end storage.
U.S. Pat. No. 5,276,860 describes a system that employs both HSM and backup technology using removable storage media like optical disks and magnetic tape. HSM is used as a means of reducing the amount of data that had to be committed to backups since data that is staged out to an optical disk is considered backed-up. However a need remains for data protection systems that provide timely and cost-effective disaster recovery and/or data migration.